Improve Statement Splitter

[psteinroe]

The statement splitter is a crate that receives sql source and returns ranges for every statement in the file. It's supposed to work with incomplete and incorrect statements. Right now, the implementation is very simple and just looks at the first n tokens of a statement to check if a new one starts at the current cursor position.

This is very fragile, and should be improved. My current idea is to instead of just using the first n tokens, allow any tokens to be part of the declaration, so that e.g. the create function syntax

CREATE [ OR REPLACE ] [ CONSTRAINT ] TRIGGER name { BEFORE | AFTER | INSTEAD OF } { event [ OR ... ] }
    ON table_name
    [ FROM referenced_table_name ]
    [ NOT DEFERRABLE | [ DEFERRABLE ] [ INITIALLY IMMEDIATE | INITIALLY DEFERRED ] ]
    [ REFERENCING { { OLD | NEW } TABLE [ AS ] transition_relation_name } [ ... ] ]
    [ FOR [ EACH ] { ROW | STATEMENT } ]
    [ WHEN ( condition ) ]
    EXECUTE { FUNCTION | PROCEDURE } function_name ( arguments )

could be defined in the parser as

CREATE [ OR REPLACE ] [ CONSTRAINT ] TRIGGER <string> { BEFORE | AFTER | INSTEAD OF } <any-tokens> ON <string> <any-tokens> EXECUTE { FUNCTION | PROCEDURE } <string> ( arguments )

we would need the following definitions

• a specific token
• optional token
• one of token
• any string token
• any number of tokens

[aleclarson]

I see you've already put a lot of work into this, but I'm curious why you don't split statements with libpg_query?

• split with scanner: https://github.com/pganalyze/libpg_query/blob/43bad3cbcd1a70a30494b64f464c3f60579884ed/src/pg_query_split.c#L10
• split with parser: https://github.com/pganalyze/libpg_query/blob/43bad3cbcd1a70a30494b64f464c3f60579884ed/src/pg_query_split.c#L165