Re: [Texmacs-dev] string-tokenize

[Michael Klein]

Hello Henri,

I did find this function tmconcat-tokenize-math that gives me the tokens I want.

Best regards
Michael

2007/7/24, Henri Lesourd < address@hidden>:

Michael Klein wrote:

> Hello List,
>
> is there a procedure in scheme which tokenizes a string into
> substrings at several characters and not deleting these?
>
Don't know what you mean by "and not deleting these". Either you
tokenize, or either you don't (?)...

The following routine splits a string at the positions where a given
character occurs :
[[
(define nest #t)
(define (string-tok s c)
   (define resu '())
   (define str '())
   (define nest-level 0)
   (define (traverse l)
      (if (or (null? l)
              (and
                 (or (not nest)
                     (eq? nest-level 0))
                 (eq? (car l) c))
          )
          (begin
             (set! resu (cons (list->string (reverse str)) resu))
             (set! str '())
             (if (not (null? l))
                 (traverse (cdr l)))
          )
          (begin
             (if nest (begin
                (if (eq? (car l) #\()
                    (set! nest-level (+ nest-level 1)))
                (if (eq? (car l) #\))
                    (set! nest-level (- nest-level 1)))
             ))
             (set! str (cons (car l) str))
             (traverse (cdr l))
          )
      )
   )
   (set! nest-level 0)
   (traverse (string->list s))
   (reverse resu))
]]

, and the following performs the reverse operation :
[[
(define (string-untok l sep)
  (define s "")
  (define (traverse l)
     (if (null? l)
         s
         (begin
            (if (equal? s "")
                (set! s (car l))
                (set! s (string-append s sep (car l))))
            (traverse (cdr l))))
  )
  (if (or (not (pair? l))
          (null? l)
      )
      s
      (traverse l)))
]]


I don't know if it is exactly what you want, but you can at least
use this as a starting point.


Best, Henri