# {py:mod}`dawsonia.ml.data.generator`

```{py:module} dawsonia.ml.data.generator
```

```{autodoc2-docstring} dawsonia.ml.data.generator
:allowtitles:
```

## Module Contents

### Classes

````{list-table}
:class: autosummary longtable
:align: left

* - {py:obj}`DataGenerator <dawsonia.ml.data.generator.DataGenerator>`
  - ```{autodoc2-docstring} dawsonia.ml.data.generator.DataGenerator
    :summary:
    ```
* - {py:obj}`Tokenizer <dawsonia.ml.data.generator.Tokenizer>`
  - ```{autodoc2-docstring} dawsonia.ml.data.generator.Tokenizer
    :summary:
    ```
````

### API

`````{py:class} DataGenerator(source: str, batch_size: int, charset: str, max_text_length: int, stream: bool = False)
:canonical: dawsonia.ml.data.generator.DataGenerator

```{autodoc2-docstring} dawsonia.ml.data.generator.DataGenerator
```

```{rubric} Initialization
```

```{autodoc2-docstring} dawsonia.ml.data.generator.DataGenerator.__init__
```

````{py:method} next_train_batch() -> typing.Iterator[tuple[numpy.typing.NDArray, numpy.typing.NDArray]]
:canonical: dawsonia.ml.data.generator.DataGenerator.next_train_batch

```{autodoc2-docstring} dawsonia.ml.data.generator.DataGenerator.next_train_batch
```

````

````{py:method} next_valid_batch() -> typing.Iterator[tuple[numpy.typing.NDArray, numpy.typing.NDArray]]
:canonical: dawsonia.ml.data.generator.DataGenerator.next_valid_batch

```{autodoc2-docstring} dawsonia.ml.data.generator.DataGenerator.next_valid_batch
```

````

````{py:method} next_test_batch()
:canonical: dawsonia.ml.data.generator.DataGenerator.next_test_batch

```{autodoc2-docstring} dawsonia.ml.data.generator.DataGenerator.next_test_batch
```

````

`````

`````{py:class} Tokenizer(chars, max_text_length=128)
:canonical: dawsonia.ml.data.generator.Tokenizer

```{autodoc2-docstring} dawsonia.ml.data.generator.Tokenizer
```

```{rubric} Initialization
```

```{autodoc2-docstring} dawsonia.ml.data.generator.Tokenizer.__init__
```

````{py:method} encode(text)
:canonical: dawsonia.ml.data.generator.Tokenizer.encode

```{autodoc2-docstring} dawsonia.ml.data.generator.Tokenizer.encode
```

````

````{py:method} decode(text)
:canonical: dawsonia.ml.data.generator.Tokenizer.decode

```{autodoc2-docstring} dawsonia.ml.data.generator.Tokenizer.decode
```

````

````{py:method} remove_tokens(text)
:canonical: dawsonia.ml.data.generator.Tokenizer.remove_tokens

```{autodoc2-docstring} dawsonia.ml.data.generator.Tokenizer.remove_tokens
```

````

`````