Create scanner for removing comments and continuations (#2230)
* Create scanner for removing comments and continuations
* When reading the requirements txt, handle line
continuations and remove comments
* Fix fast_generator test
* Update pkg/requirements/requirements_test.go
Co-authored-by: Aron Carroll <aron@replicate.com>
Signed-off-by: Will Sackfield <will.sackfield@gmail.com>
* Fix config test
* Include beginning of line comment testing
* Handle CRLF line breaks
* Move no whitespace string logic to scanner
* Restore removing whitespace strings above scanner
* Scanner exits when the token is nil on return
---------
Signed-off-by: Will Sackfield <will.sackfield@gmail.com>
Co-authored-by: Aron Carroll <aron@replicate.com>
