New concept for eliminating compression (280 byte limit)

[KilledByAPixel]

I've put some thought into this over the 1000 or so dweets I've published and here's my suggestion, it's very simple.

Change the limit to 280 bytes. There are many benefits to this...

• Since bytes are counted rather then characters, unicode character compression would no longer be necessary.
• Makes it much easier to newcomers to understand what is going on.
• Legacy dweets are still be allowed, most will probably be below 280 bytes.
• Special category/icon for dweets that are 140 bytes or less called "half dweets".
• All current uncompressed dweets would fall into the category of half-dweet.
• You can sort by full or half dweets.
• Breathe some new life into dwitter, let's see what else is possible with more code.
• New challenges to fit things without using compression, maybe there are new tricks we can discover.
• Get more exposure for dwitter because we would be able to make cooler stuff.
• We can still keep the uncompress button to show up for legacy dweets. Or maybe it would make more sense to just automatically uncompress them.

That's my idea! It's pretty simple, just a change code length to be counted by bytes and up the limit to 280 (needs to be changed on both front and back end), old dweets will still work.

[ThomasBrierley]

Hi Frank!

I too am interested to see "double dweets" discussed again, and am glad you're the person to resurface it.

I have quite a few thoughts on this, so bear with me 😬

Breathe some new life into dwitter, let's see what else is possible with more code.

I think this may be the strongest argument... 140 Unicode / 194 ASCII has been amazing, but it does feel harder and harder to find new and interesting things to squeeze into it. I'm not completely sure if this is a personal limitation, a temporary plateau, or a fundamental combinatorial limit of 140 characters, but I'm leaning toward the latter (yes there are 128**140 = x10^295 ASCII combinations, but there is a much smaller subset of JS that actually runs, a smaller subset that results in pixels, and an even smaller subset that is interesting and unique). 280 would not be merely doubling the combinations, it would be multiplying it by 128**140! so we are unlikely to stagnate in our lifetime... but it's also not so huge as to feel meaningless.

The other consideration is the arbitrarity of these limits, we have to pick something, but we should pick one thing... multiple or variable limits feels too arbitrary, because it's no longer a shared limitation. I do believe that the singular limit of Dwitter is part of what makes it so compelling: "do anything on a canvas with JS, in 140 characters", that's it... you know what to do... and you instantly appreciate and understand what has been done.

My point is, while I like the idea of moving onto double dweets, I also think maintaining multiple limits is a bad idea... so this proposal should be a change, not an addition. Though something along the lines of a badge of appreciation for the original 140 limit is a good idea... There is already a kind of understanding and appreciation for uncompressed dweets, <140 dweets.

Since bytes are counted rather then characters, unicode character compression would no longer be necessary.

This is tricky. As soon as we start talking about bytes outside of ASCII and enter the land of Unicode, we have to start talking about encodings, and different encodings consume bytes in different ways. As far as JS is concerned I think we only need to consider two:

Modern text files today are stored in UTF-8, and most databases use utf8mb3 or 4 for Unicode which is roughly equivalent. Both of these use one byte for the ASCII part of Unicode, then variable (multi-byte) bytes for extended Unicode points up to 4. To differentiate from single byte ASCII these come with special prefixes indicating the byte length and prefixing continuation bytes, I wont bother explaining in detail because wikipedia does a pretty good job. The point is that these prefixes take up codepoint space, so the first 2**16 unicode points do not fit inside 2 bytes with UTF-8 unlike other encodings.

Internally JS uses USC-2 which is essentially UTF-16 (don't ask me the difference, i'm not sure anyone can give you a straight answer on this either). UTF-16 is not a variable byte encoding, it stores everything in 16 bit words as the name implies: ASCII? still two bytes, unicode <2**16? two bytes, unicode >2**16? now you've got "surrogate pairs" 2*16bit words, with a bit of codespace gone to prefixing for surrogate pairs (I lied the boundary is a bit less than 2**16).

When you compare these side by side UTF-16 looks a bit silly for most of the chars we care about (ASCII), because it will double the byte count, but we could probably discount this as it's only an internal representation (in fact I think this is how most JS engines counts string length, ignoring empty upper bytes). However there are more subtle differences for the codepoints above ASCII, where some will fit into 2 bytes in UTF-16 but will need 3 bytes for UTF-8.

I'm not 100% sure how the differences turn out empirically when looking at UTF-8 and JS string.length side by side, but i'm pretty sure there will be differences, (there are even differences between JS engines for string.length).

... anyway, I find it hard to understand and explain all these subtleties even though I know they exist 😕 so maybe you can see why i'm not convinced we are going to be able to explain them to people who don't even know they exist! (unless we stick with one byte ASCII and say goodbye to emojis and unicode byte packing).

[edit]

I just realised JS string.length is supposed to count UTF-16 words, which is why every character that fits into one UTF-16 word =1char and characters needing a surrogate pair are counted as 2 chars... anyway, string.length!== bytes in general, but it does == 1/2 the bytes of UTF-16 encoding.

[edit]

alternatively... we just say bytes are counted in UTF-8 which has basically become the universal storage encoding for text, even though it's not used internally for programs and languages for performance reasons. But "storage" is probably more tangable, a tweet on twitter is no doubt stored as utf8mb4 on a database, a tweet copied into a text file is no doubt saved as UTF-8 on the block device etc etc.

[lionleaf]

Thanks for resurfacing this discussion! I know it's been simmering in the background and it's clear you've spent a lot of time reflecting on this. It's also great to get this input from one of the most prolific dweeters!

Here's my personal stance at the moment:

• I agree 280 limit would make sense!
• The exact details on how to count is a separate, but important, discussion. I lean in favor of something that is relatively easy to understand in most circumstances when sticking to ASCII (i.e. 1 ascii == 1 count)
• Great points on breathing new life into dwitter; a lot of new unexplored techniques!
• Yes to only having a single active rule set; keeping things focused is valuable.
• As for legacy dweets, I think something like the badge you suggest makes sense.
• I also think we could move to a purer JS, by removing the S / C etc shorthands. This breaks legacy dweets, so we would have to render them in a different way, which is fine. Breaking backward compatibility allows us to reconsider the full "language"
• Even if you count bytes you won't eliminate compression; yes the byte packing tricks of dwitter stop working, but with 280 limit there's more room for compression bootstrapping code, so I bet someone could still find some clever schemes; it's easy enough to make dweets with a subset of the characters from ASCII. But this sounds like a fun challenge for the community; and if there's a new default compression technique that becomes popular we can add tools/detection for it.

All points are open for discussion, but that's where my head is at the moment. Thoughts?

As for implementation; I'd like to get the new frontend across the finish line first, and I'm starting a new job on Monday so the near future is hard to predict. Let's keep the discussion going here and on discord 👍

[ThomasBrierley]

The exact details on how to count is a separate, but important, discussion. I lean in favor of something that is relatively easy to understand in most circumstances when sticking to ASCII (i.e. 1 ascii == 1 count)

Agreed. This fact alone means that If bytes are to be the new metric, the encoding must be UTF-8, because counting bytes in any other unicode encodings will result in more for ASCII. Which means we are back to two options:

• UTF-8 Byte Counting (Proposed): Count UTF-8 encoded bytes, ASCII = 1, other = 2/3/4
• Codepoint Counting (Existing): Count unicode, each code-point (character) = 1

Sticking to codepoint counting would continue to allow the ASCII unicode packing via UTF-16 surrogates, so 280 = 2 * (280 - 43) = 474 ASCII characters which is a 169% boost, compared to 194 which is only a 138% boost, due to the constant overhead... So pretty sure we don't want this?

The more I think about UTF-8, the more it makes sense. Demos are usually limited in bytes but also by stored size, not size in memory. Text in a modern DB is UTF-8, text in a HTTP request is usually UTF-8 encoded, text in a file is usually UTF-8 encoded, pretty much everything that is not an internal representation for performance is UTF-8... Not to mention pointing at and explaining UTF-8 encoding is way simpler than the headache of UTF-16/USC-2...

https://en.wikipedia.org/wiki/UTF-8#Encoding

Sorry for the detour.

As for implementation; I'd like to get the new frontend across the finish line first, and I'm starting a new job on Monday so the near future is hard to predict. Let's keep the discussion going here and on discord +1

Understood, no rush whatsoever, and ultimately this is your decision 👍 I'm just glad to see it being discussed. Hope everything goes smoothly for you on Monday.

[edit]

Quick compatibility check... existing "compressed" dweets should be 43 bytes for the ASCII unpacking code + a max of 4 bytes for each of the 97 packed characters when using UTF-8 encoding *4 = 237 bytes max. Which almost achieves what @KilledByAPixel suggests (eliminating compression) by bringing it close to 280 bytes.

But we should eventually check this empirically for cases outside of the usual compression methods.

[KilledByAPixel]

Hi, a few things...

I also think we could move to a purer JS, by removing the S / C etc shorthands. This breaks legacy dweets, so we would have to render them in a different way, which is fine. Breaking backward compatibility allows us to reconsider the full "language"

It's a great idea but after thinking about it there is another problem, remixing legacy dweets. If someone tries to remix a dweet that uses those shortcuts, how does it work? Even just looking at a legacy dweet is going to be confusing for new users unaware of these shortcuts.

Personally I don't mind the shortcuts as much as the compression. I think a few shortcuts are friendly to newcomers while compression is scary. It would be cool to have vanilla js, but it's really just 4 quick defines. I also appreciate the time savings, i've typed S and C many thousands of times over.

I'd actually suggest one additional shortcut to be introduced, nothing crazy mind you...

H - hsla color, works the same as R but for hsl. This isn't really gonna change what we can do math wise but should result in better color usage. Generative art generally is easier to produce and looks better in hsl.

HSL would be an nice addition whether or not we decide to go to 280 bytes.

Quick compatibility check... existing "compressed" dweets should be 43 bytes for the ASCII unpacking code + a max of 4 bytes for each of the 97 packed characters when using UTF-8 encoding *4 = 237 bytes max. Which almost achieves what @KilledByAPixel suggests (eliminating compression) by bringing it close to 280 bytes.

The way I envision it is like right now, we already added code to auto uncompress dweets. So, compressed dweets would be swapped out with uncompressed code and show the uncompressed byte count which for most is capped at 194. There can still be a button or something to view the original uncompressed code, but we don't even need to allow editing that, it should be read only.

It would be great if we stored the uncompressed version of legacy dweets in the database along side them. I can write a script to spin through the database pump out json of all the uncompressed code. I've been meaning to do that for dweetabase so compressed dweets are searchable.

There are some older dweets that are compressed in different ways that may be above 280, but that's pretty rare. In this case it would show the byte count in red like it currently does.

[ThomasBrierley]

Ah i see! So the change could achieve multiple things... we get more characters to play with, the existing uncode packed dweets can be made more accessible by simply replacing them with their 194 byte ASCII, obviating the compression toggle idea... and a new ASCII unicode packer will not emerge because now we are counting UTF-8.

However other more interesting compression techniques might continue and emerge that don't obfuscate the source so much.

This feels like a really cohesive change.

[KilledByAPixel]

You got it exactly.

The problem I have with the current compression is it is dumb, because it doesn't actually compress anything. I mean it's a brilliant invention but completely useless outside of dwitter. It silly that compressed dweets take up more space.

I would love to see people experiment with real compression schemes that are case dependent and potentially useful for other size coding projects. Like for example my way of drawing bitmaps on dwitter will no longer work as it uses a similar unicode trick. So, we'd need to find a new way to pack images.

[lionleaf]

The more I think about UTF-8, the more it makes sense.

Sounds sensible to me!

It's a great idea, but after thinking about it there is another problem, remixing legacy dweets. If someone tries to remix a dweet that uses those shortcuts, how does it work?
You raise a valid point, but I'm still not sure we need backward compatibility. As for remixes, we could either disable it, or auto-expand it (which could now be >280 in some cases). It's also possible to have a legacy-remix-mode, but that gets us back to supporting multiple modes.

Another alternative is to add a preamble when remixing since there's now more space:

S=Math.sin;
C=Math.cos;
T=Math.tan;
function R(r,g,b,a){a=a===undefined?1:a;return "rgba("+(r|0)+","+(g|0)+","+(b|0)+","+a+")";}
$REMIX_CODE

Keeping c and x makes sense, though.

I'd actually suggest one additional shortcut to be introduced, nothing crazy mind you...

H - hsla color, works the same as R but for hsl.

Yes, we should revisit the shortcuts; if we're not removing them all, we could certainly consider whether we should add some. Note that this could also break old dweets; if they rely on H being undefined, for instance.

[KilledByAPixel]

Another alternative is to add a preamble when remixing since there's now more space:

I don't think that would work by just adding it at the top because so many of the best dweets rely on those shortcuts being setup before the dweet runs. Not as trig functions, but because they are valid variables. It is not a small amount but in the hundreds that use techniques relying on that.

Can you explain more your reasoning about wanting to go vanilla?

Note that this could also break old dweets; if they rely on H being undefined, for instance.

Maybe, but it is unlikely, I would be surprised if it came up. There are some dweets that use the variable H but relying on something being undefined is very unusual for a dweet.

[lionleaf]

Can you explain more your reasoning about wanting to go vanilla?

Since the rulesets are already arbitrary, my bias is to stay as close to vanilla js as possible. Makes it somewhat clearer what's going on for people who know js. But I'll admit there's an allure to the purity itself. It's also a much more consistent approach than adding high-value shortcuts for a semi-arbitrary subset of js. And I think 280 would be enough to make up for the loss of the shortcuts.
Again, I'm not set on this and am open to discussion, so happy to keep exploring the implications.
Having S/C/T shortcuts certainly makes math relatively cheaper, so maybe removing them will be a net negative as it reduces the maximum mathematical complexity possible?

Note that this could also break old dweets; if they rely on H being undefined, for instance.

There are some dweets that use the variable H but relying on something being undefined is very unusual for a dweet.

The scenario that came to mind was variables that are initialized once, by relying on them being undefined for the first iteration. But you're right it's probably not many dweets that would break if we add a few more shortcuts, just mentioning that theoretically there could be.

[Lerc]

I agree with counting by bytes.

When it came to utility functions. Either extend or eliminate. I like either idea. Extending exposes us to feature creep, but that may not necessarily be a bad thing.

If utility functions were added at a slow steady pace, say one per month by vote from a list of candidates, it would allow for the growth of a well considered micro library that might be quite nice to use in general. It would mean an ever growing capability of Dweets, but it would also mean committing to non-vanilla JS. An extending library might require versioning and Dweets being tied to a version that existed at post time to avoid compatibility. It may not be an issue if no Dweets rely on things being undefined at call time.

Taking the path of Vanilla JS I also had the idea of a default preamble of

M=Math;S=M.sin;C=M.cos;T=Math.tan;R=(r,g,b,a=1)=>"rgba("+[r|0,g|0,b|0,a]+")

note, this version also defines M.

Include the preamble for old Dweets, For new Dweets just prepend it to the preamble to the default and let people reuse the characters

[KilledByAPixel]

If you do get rid of the shortcuts, I think it would make sense to get rid of some other variables too like frame, FPS or anything else that pollutes the space. Though I still lean towards keeping everything as it is now.

[ThomasBrierley]

Like for example my way of drawing bitmaps on dwitter will no longer work as it uses a similar unicode trick. So, we'd need to find a new way to pack images.

I'm pretty sure that's not the case. You can still stuff bits into a unicode point however you want... how many bytes it ends up as is determined by UTF-8 encoding, but how it's used in JS is completely independent. It will certainly count for more when packing beyond 8bits with byte counting, but it's still an efficient way of encoding bitmaps etc in JS source and there's no reason it should stop working.

If you do get rid of the shortcuts, I think it would make sense to get rid of some other variables too like frame, FPS or anything else that pollutes the space. Though I still lean towards keeping everything as it is now.

I'm two minds on this, on the one hand I feel like it will be a can of worms for compatibility after having attempted to make changes to the dweet.html file myself. On the other hand I feel like that file is way more complex than necessary and agree the scope could be a lot cleaner... I did an experiment a while ago removing all of the postmessage stuff and external libraries and found that the only significantly complex thing that was actually necessary to keep was the instrumentation, to prevent infinite loops when you are mid typing something... I dislike the existing solution of injecting code because it requires a whole parser and is quite messy, so found two other methods:

1. Offscreen canvas context with webworkers, this is unfortunately chrome only at the moment, but is the ideal solution because you essentially get a separate thread but with a canvas API... this is great because it's also non-blocking for the UI, but more to the point you can monitor the worker's progress on a frame from the main thread, so you can intervene and just kill it without having to inject code.

2. Intercept all method calls and property setters on the canvas context instance. This is not so ideal, but it catches 90% of cases without injecting code, the other cases are where a loop is being constructed that doesn't yet have any calls to the canvas context, or separate loops e.g a raymarcher, that doesn't call canvas methods at all. This one has quite a few holes obvously, so I'm hoping the offscreen canvas is implemented in Firefox and Safari sometime in the future.

[Lerc]

If you do get rid of the shortcuts, I think it would make sense to get rid of some other variables too like frame, FPS or anything else that pollutes the space. Though I still lean towards keeping everything as it is now.

You could potentially put the essentials into parameters and end up with a global free

function(c,x,t) {
  ...dweet body...
}

This would break a few dweets, but those ones were a bit sus' anyway :-)

Amusingly, if you passed x,c,t as parameters. that would allow you to have all dweets as functions of the form dweet24232(c,x,t) and call dweets from your dweets. You could do a prescan to only bring in dweets actually called. I would also like to make clear that I'm not suggesting we actualy do this thing, that way madness lies, but imagine the possibilities.

[lionleaf]

When it came to utility functions. Either extend or eliminate. I like either idea. Extending exposes us to feature creep, but that may not necessarily be a bad thing.

If utility functions were added at a slow steady pace [...] An extending library might require versioning and Dweets being tied to a version that existed at post time to avoid compatibility.

While a fun idea, and something that I think would be fun for the regular community, I do not want to go in the direction of an extending library for two reasons: As you said, the increased cost of backward compatibility seems like a potential pain, as dweets behave in strange ways. But more importantly, this creates an increasingly complex "dwitter language" that adds to the confusion of new users. I like the premise to be as simple as possible.

So either a small well-selected set of shorthands or none :) If we want to play with evolving rules I think a potential idea would be "seasons" were do a full change of the ruleset, breaking backward compatibility, every year or two.

If you do get rid of the shortcuts, I think it would make sense to get rid of some other variables too like frame, FPS or anything else that pollutes the space. Though I still lean towards keeping everything as it is now.

You could potentially put the essentials into parameters and end up with a global free

function(c,x,t) {
  ...dweet body...
}

I like this idea, only a handful of parameters to the function. Also, if we think 280 without shorthands doesn't allow enough complexity, there's nothing stopping us from doing 310 or something to account for it. In my mind adding shorthands is similar to increasing the limit, while also biasing the cost of functions in a different direction than vanilla js.

[...] can of worms [...] dweet.html file [...]

Yes absolutely. Having a clear separation of legacy dweets will allow us to serve the old dweet.html file for those dweets, ensuring they will still run, while starting with a clean slate!

injecting code [...] is quite messy, so found two other methods:

1. Offscreen canvas context with webworkers [...]
2. Intercept all method calls and property setters on the canvas context instance. [...]

Very cool! Webworkers would really be the solution, last time I looked at it there was lacking support and potential workarounds would have horrendous performance. Worth revisiting at some point!