i jus wanted to get dis outta my system >v< ...
i dun like those boring linear model structures... they work... bt they dun look fun, nor intuitive. they jus produce output... which is boring!
pls, if some researcher with lotsa gpus sees this, maybsies try this kinda architecture... u dont evn have to credit me, just try it out n see where it goes ~ ~ ~
Afaik all LLMs have very derp recurrance, as that's what provides their context window size.
The more recurrant params they have, the more context window they can store.