Perhaps the time isn't ripe for this, and perhaps it never will be, but
UTF8 seems to be handled by just about everything these days. I suspect
this is a crazy suggestion, but on the other hand perhaps people looking
back from 2100 will think "It was crazy that they stuck exclusively with
ASCII syntax characters for so long after Unicode was widely available".
Sometimes when you have quite a few levels of brackets, and there are more
than one of the same type, might it be better to allow variants of each
type of bracket character? Unicode provides bold, double, small,
superscript, subscript, and white-filled (hollow) variants of round, square
and curly brackets, and top and bottom ticked variants of square brackets.
Perhaps not enough platforms will be able to display them? And entering
them may be fiddly, although programmable editors and IDEs could let you
type the standard characters but pick variants on a round-robin basis
within each expression.
But it might be handled better in the display in editors and IDEs (perhaps
syntax coloring brackets by their depth). Or some might say not to write
such deeply nested bracketed expressions.
Implementation parsers could simply translate all the brackets to the base
types, or they could treat them as equivalent but distinct, and check that
the open and close brackets match, which might catch a few errors.
There are also other quote characters available, such as the guillemets
traditionally used in French. There's absolutely no need to use such
things, but for people working on code which will be used internally in,
say, French or Quebecois organizations might welcome it.
OK, shoot this one down now :-)