Soft deprecation of the project?

@kddnewton

So, Prism is out and it's:

correct
comes out of the box with MRI (and IIRC with other Ruby implementations)
faster (I think on TruffleRuby/JRuby it's absurdly faster because it doesn't run in interpreted mode like it does with parser)
always up-to-date
written in a language with perfect interop capabilities

On the other side backporting changes to parser becomes more and more difficult. MRI doesn't use yacc/bison anymore, instead there's Lrama that is also an LALR(1) parser generator but it has more "syntax sugar" that Racc doesn't have, and so the grammar will diverge more and more over time. And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I personally believe that parser should be soft-deprecated. What do you guys think? @kddnewton @bbatsov @koic @palkan @marcandre @mbj

And if you agree do you see anything we could do to parser to make transition easier?

The real use case of this gem left for me is multi version capability. Prism is not shipped with all the ruby versions my use cases still have to support, basically all non EOL (MRI) rubies.

If this gem where to stop supporting future ruby versions I'd be fine. Cannot speak for the rest of the ecosystem OFC.

In general I think this is the right long-term direction, but I'm uncomfortable soft deprecating until we mirror the tree rewriting functionality, since so many consumers use that. I think it could be added to the Ruby API of Prism relatively easily, but I haven't had time to look into this yet.

I'm not sure if there are any other APIs that I'm not thinking about, but that's the main one I know we are missing.

Indeed. I don't see how we can deprecated parser without the rewriting functionality. RuboCop also relies on the multi-version capability quite a lot, although I guess we can live without it if push comes to shove.

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases? I think that a realistic time-frame for full migration to Prism is probably 2-3 years (for the projects that are still active, that is), so parser will definitely require some maintenance is the mean time.

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases?

My initial thought was to stop doing it. At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

I think we need to put evolutionary pressure on and NOT release compatibility with new versions.

I'm not super fond of measures that will be quite disruptive to the end users - to me it seems just wild to stop supporting something without providing a complete alternative for it. E.g. what happens if the rewriting is still not supported by Prism when Ruby 3.5 comes out?

At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

Breaking existing apps is not exactly in the service of the community either. ;-)

I'm not arguing that parser should go away - I'm just concerned about abandoning the project prematurely and causing pain for the users of tool depending on it.

To play a bit of devils advocate here: How is not supporting a new future ruby release "breaking existing apps"?

Okay, "causing problems for the existing apps". I'm fairly certain RuboCop's users won't be exactly happy if we tell them we don't really support newer Ruby releases for whatever reasons. :-) And I'm not sure that simply stopping work on parser will motivate everyone to implement whatever is missing, etc. Also - how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

Moving fully to Prism's native AST is going to be quite the time investment for many projects. I'd like for us to get there, but ideally I don't want us to be pressured into a timeline for this given that everyone working on RuboCop does this in their spare time.

Is there a translation layer from prosm to this parsers AST somewhere?

Is there a translation layer from prism to this parsers AST somewhere?

Prism supports AST translation for popular parsers: https://github.com/ruby/prism/tree/main/lib/prism/translation

And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I think, deprecating Parser as soon as parse.y is deprecated makes sense. Otherwise the effort required to port Ruby changes to Parser would be enormous.

how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

From my experience, adding new nodes could be done relatively easy (compared to upgrading the grammar and lexer). I see, how, in theory, we can use Prism for newer Ruby versions as a parser and a Parser-compatible AST builder.

So, if we can keep Parser Ruby internals and APIs work in new Ruby versions (without providing the corresponding syntax support), that would smooth the transition.

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

Yes, that's more or less what I proposed (probably the word "deprecate" was a bit misleading). I'm not proposing to yank parser from rubygems.org, only to stop backporting new grammar changes.

I personally never touched rewriting API, shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Breaking existing apps is not exactly in the service of the community either. ;-)

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Yeah, it works right now. Well, at least, for Ruby Next.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

Is there an unparser equivalent yet?

Truthfully I don't know if parse.y is going to be deprecated, or if that will happen any time soon at all. I imagine it could stick around for quite a long time. That being said, it is getting further away from strict bison/racc syntax, so I know it's quite a maintenance burden to hold on to.

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

I think this would work, but we will want to test the translation layer even more. I think there are issues with string escapes at the moment, but I'm not sure.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

That would be really great.

Is there an unparser equivalent yet?

I haven't managed to get it over the finish line yet, but yes https://github.com/ruby-syntax-tree/syntax_tree-prism is that work. It's almost done.

@koic

Is there an unparser equivalent yet?

Probably no, but I've found "untokenizer" made by @koic 😄

@whitequark

@whitequark How about me and @koic?

(just a note - he has a different handle there https://rubygems.org/profiles/koic_ito)

@bbatsov

@bbatsov Thank you. I was previously granted release privileges along with commit privileges in #608 (comment).

@koic

@koic So I guess you can help with the list of PRs that @Earlopain submitted and I'd like for us to move along somehow. I think he's also a good candidate for a co-maintainer for parser.

@bbatsov

@bbatsov You have the commit bit and the gem publish rights.

@iliabylich

Yeah, I can see the currently open PRs, but I'm starting to have some concerns about whether maintaining both Parser and Prism::Translation::Parser is truly the best approach.

As you know, starting with Ruby 3.4, Prism has become the standard parser for Ruby, and it will likely continue to support any new syntax introduced in Ruby 3.5 and beyond. When that happens, Prism::Translation::Parser will need to be updated accordingly.

On the other hand, as @iliabylich mentioned, the parse.y files used by (alternative) CRuby’s parser generator Lrama and the one used in the Parser gem with Racc are diverging.

At this point, while it is not impossible to keep tracking CRuby’s parse.y, I am still uncertain whether the cost is justified.

As a result, the implementation of the it parameter syntax in Ruby 3.4 has also been opened a PR precede in Prism::Translation::Parser, as it has been easier to support there:
ruby/prism#3481

As shown in the PR, the AST designed by the Parser gem is highly compatible with RuboCop's needs, and Parser::Translation::Parser is positioned as an extension that advances ahead of it.

Prism (or Lrama) was intended to simplify parser maintenance with a Universal Parser, yet the current reality involves maintaining multiple parsers. How to interpret this situation is under consideration. Of course, no decision has been made at this point.

@koic

@koic Long-term - no arguments from me. But I'm guessing it'd be prudent to maintain parser to some degree in the short term (e.g. for Ruby 3.4 and 3.5) until more people have switched to Prism.

@Earlopain

Also it seems to me that @Earlopain's improvements are non-controversial and if the work is done we might as well ship it. :-)

I think he's also a good candidate for a co-maintainer for parser

Thanks for your word of confidence, I woudn't mind it.

I made the PRs after I saw this issue here to see how much effort it would be to make it fully compatible with 3.4. I quickly realized that no, I won't be able to make that work. I wouldn't mind seeing the bugfixes merged since I already put in the work.

The ones specific to 3.4 syntax, merging them would give a wrong impression about intentions to support it in the future, which I don't want to give anymore. In my opinion, these should be closed.

The ones specific to 3.4 syntax, merging them would give a wrong impression about intentions to support it in the future, which I don't want to give anymore. In my opinion, these should be closed.

Got it. I was hoping that those were indication that it supporting 3.4 would be doable. Anyways, I guess we'll focus on getting the bugfixes out, as they would benefit everyone.

I'll update the README here accordingly when I get the chance (e.g. soft deprecation, etc) and we'll need to be clearer in https://docs.rubocop.org/rubocop/compatibility.html about the need for people to use Prism if they need to target Ruby 3.4+.

Got it. I was hoping that those were indication that it supporting 3.4 would be doable

Surely it is for someone with more expertise in ragel and parsers in general but I am not that person. All my changes were possible in ruby without needing to touch the syntax directly (which looks daunting to say the least).

I'll update the README here accordingly when I get the chance (e.g. soft deprecation, etc) and we'll need to be clearer in https://docs.rubocop.org/rubocop/compatibility.html about the need for people to use Prism if they need to target Ruby 3.4+.

I don't think there's a need for that. Currently, it works very fine with parser because the syntax barely changed and once ruby/prism#3481 is merged and released rubocop-ast can just hard-depend on prism and simply make that choice for the user without anyone needing to explicitly opt in. It can only parse Ruby starting from 3.3 but has no trouble running on 2.7 which works out very well for RuboCop in that regard

I have also been thinking about https://docs.rubocop.org/rubocop/compatibility.html page. At present, for most users, there is no significant difference between the Parser gem and Prism::Translation::Parser. From a compatibility standpoint, the Parser gem has a slight advantage.

However, with the next release of the Prism gem, reported incompatibilities in Prism::Translation::Parser are expected to be improved. Furthermore, once it block parameter syntax support from ruby/prism#3481 is included, Prism::Translation::Parser will provide more modern support for Ruby 3.4 users.

That will likely be one of the key points for making a thinking for RuboCop's backend parser.

chore: switch from parser to prism

Support `Prism::Translation::Parser35` for Ruby 3.5 parser

chore: switch from parser to prism

Navigation Menu

Repository navigation

Soft deprecation of the project? #1046

9 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Soft deprecation of the project? #1046

Description

Activity

mbj commented on Nov 21, 2024

kddnewton commented on Nov 21, 2024

bbatsov commented on Nov 21, 2024

iliabylich commented on Nov 21, 2024

mbj commented on Nov 21, 2024

bbatsov commented on Nov 21, 2024

mbj commented on Nov 21, 2024

bbatsov commented on Nov 21, 2024

mbj commented on Nov 21, 2024

palkan commented on Nov 21, 2024

iliabylich commented on Nov 21, 2024

palkan commented on Nov 21, 2024

mbj commented on Nov 21, 2024

kddnewton commented on Nov 21, 2024

iliabylich commented on Nov 21, 2024

9 remaining items

bbatsov commented on Feb 27, 2025

bbatsov commented on Feb 27, 2025

koic commented on Feb 27, 2025

bbatsov commented on Feb 27, 2025

whitequark commented on Feb 27, 2025

koic commented on Feb 27, 2025

bbatsov commented on Feb 27, 2025

bbatsov commented on Feb 27, 2025

Earlopain commented on Feb 27, 2025

bbatsov commented on Feb 27, 2025

Earlopain commented on Feb 27, 2025

koic commented on Feb 27, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions