GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models

Geez, that's really harsh.

I don't think any single thing you've claimed is factually wrong, and I don't speak for Eleuther nor am I attempting to justify their claims.

But.

As I understand it (mostly from lurking on their discord and reading publicly available materials) this is a group of volunteer academic types trying to replicate something great and awesome, with the only goal of giving it to the world. You could cut them some slack.

I can't speak for you, but as a "for free, weekend project" what they've done certainly makes me feel I need to up my game.

What I find interesting about their marketing(?) is that they identified a market niche that they want to position themselves in.

Enterprise customers that have no idea about the technical details will just hear about OpenAI's success in this fancy new model and assume that Eleuther can deliver.

I mean, most use cases for "big data" projects that are tiny in comparison with Alphabet's datasets will just work with GPT2 fine, probably.

And Enterprise customers that hear those claims and see some code, maybe some demo, is enough for them to start the consultancy process.

In my opinion that's a policy problem that OpenAI introduced by not requiring the absolute reproducability of both the code and model, and both training and dataset of their models upon release.

Stakes are pretty high in the AI industry, and OpenAI actively influences it. My dream was in the beginning that they are a source of verification, audits and "proof" that models are legit...yet I have the feeling lately that they just buzz around like everyone else.

To this date I haven't seen anyone replicate any of the DNC results, for example.

Anyways, just my two cents on this one.

Good perspective but I would like to hear the response from the developers before concluding too much.

This is not meant as a goad to you, but more just as more info for everyone, my understanding is it is an open source community of like minded people type project (as opposed to a bigco) and actively solicits contributions (by which I mean code and data) so anyone seeing room for improvement is welcome to step in from what I can tell.

I did find your comment helpful and informative; just adding another angle here.

Disclaimer: I know absolutely nothing about machine learning.

Isn't GPT-3 the architecture? Are they doing something different or why would it not scale?

Fixed title to reflect that, thanks

Yeah. They say they are doing a 10B release soon[1].

I suspect they have run into training issues since they are moving to a new repo[2]

[1] https://twitter.com/arankomatsuzaki/status/13737326468119674...

[2] https://github.com/EleutherAI/gpt-neox/

It was disappointing to see just how quickly ClosedAI changed its tune once they produced something of value.

Many people (correctly, in my view) criticized OpenAI for the name, saying that openness should be evaluated on a case by case basis. Glad they listened to critics instead of trying to maintain consistency for its own sake.

GPT-3 is a model architecture, not a model. While the largest GPT-3 model is 175B, that very paper has a table that includes "GPT-3 XL" (1.3B) and "GPT-3 2.7B" as models in the GPT-3 architecture. The 2.7B model is the same size as Ada, a model that OpenAI currently sells API access to under the moniker "GPT-3"

what does he mean when he says 1T or bust? Is he referring to 1 trillion parameters? Are you saying that GTP-3 has 2.7 trillion parameters? Does it mean that to get to GPT-3 level it needs 100x more amount of dataset?

Hi! Thanks for trying it out. There was a bug that should now be fixed. When I run the example unicorn prompt I get the follow. Don't hesitate to open an issue if you're still having trouble.

"In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

Bebek Uranzoglu, another member of the research team from the University of British Columbia, was working on a project the Latino-Canadian rodeo competition equipos to document a rare and remarkable ecosystem in the Andes Mountains.

His curiosity was piqued when he spotted an adolescent herd of about 10 unicorns foraging in a forest near the valley of the Jumbo Flu Group. The unicorns — whose numbers once swelled to 46,000 — were perched on the forest floor and watched the researchers work.

Urizoglu grew excited when he spotted another group that seemed to be thriving in an area below the herd. The team hoped the apparent population growth would indicate a human presence.

But when a team of researchers set up a camera trap, they were surprised to find the unicorns in the first place, and in a forest near a lake — in fact the forest was almost entirely made up of the animals. Despite their own magical presence, the team could not see the herd was populated by humans.

“The whole place almost smelled like animals,” says Bebek. “We were never able to find human footprints at any of the points we stood at. The trees were so large, you wouldn’t have been able to walk 40 meters through them. We assumed that the truth of the matter was, ‘Well the deer didn’t like this forest at all.’”

Same here. I managed to make it "work" in the sense that it wouldn't crash during inference, but then it generated gibberish. Has anyone managed to make it work reliably?

You may indirectly support the project by supporting the host, that hosts their data, https://the-eye.eu

Right on the front they write:

    > Hey there fans! We are currently looking for help funding large storage upgrades, 
    > if you want to help us serve more data see our donation options (crypto, etc) 
    > Thanks for reading, happy downloading!

Indirectly they say you can donate money, in the form of computation that can be rented: “As an independent organization, we are dependent upon donations for our computing costs. We are always interested in speaking with people who can donate compute times.”