Re: [Texmacs-dev] string-tokenize

[Henri Lesourd]

Michael Klein wrote:

> Hello List,
>
> is there a procedure in scheme which tokenizes a string into 
> substrings at several characters and not deleting these?
Don't know what you mean by "and not deleting these". Either you
tokenize, or either you don't (?)...

The following routine splits a string at the positions where a given
character occurs :
[[
(define nest #t)
(define (string-tok s c)
  (define resu '())
  (define str '())
  (define nest-level 0)
  (define (traverse l)
     (if (or (null? l)
             (and
                (or (not nest)
                    (eq? nest-level 0))
                (eq? (car l) c))
         )
         (begin
            (set! resu (cons (list->string (reverse str)) resu))
            (set! str '())
            (if (not (null? l))
                (traverse (cdr l)))
         )
         (begin
            (if nest (begin
               (if (eq? (car l) #\()
                   (set! nest-level (+ nest-level 1)))
               (if (eq? (car l) #\))
                   (set! nest-level (- nest-level 1)))
            ))
            (set! str (cons (car l) str))
            (traverse (cdr l))
         )
     )
  )
  (set! nest-level 0)
  (traverse (string->list s))
  (reverse resu))
]]

, and the following performs the reverse operation :
[[
(define (string-untok l sep)
 (define s "")
 (define (traverse l)
    (if (null? l)
        s
        (begin
           (if (equal? s "")
               (set! s (car l))
               (set! s (string-append s sep (car l))))
           (traverse (cdr l))))
 )
 (if (or (not (pair? l))
         (null? l)
     )
     s
     (traverse l)))
]]


I don't know if it is exactly what you want, but you can at least
use this as a starting point.


Best, Henri