How may I construct a value of vec of vecs for a record batch?

[finlaydotb]

I can create column of type "UTF8" as follows

    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Int32, false),
        Field::new("payload", DataType::Utf8, false),
    ]));

    let vec_of_strings: Vec<String> = vec!["one".to_string(), "two".to_string()];
    
    let batch = RecordBatch::try_new(
        schema,
        vec![
            Arc::new(Int32Array::from_slice([1, 2])),
            Arc::new(StringArray::from(vec_of_strings)),
        ],
    )?;

    ctx.register_batch("demo", batch)?;

Executing a query against this, like so

    let df = ctx.sql(r#"
       SELECT *
       from demo
    "#).await?;

gives the expected results

+----+---------+
| id | payload |
+----+---------+
| 1  | one       |
| 2  | two      |
+----+---------+

Now I have a usecase where the payload should be an array. So something like this

+----+---------+
| id | payload |
+----+---------+
| 1  | [piano, guitar, drums]   |
| 2  | [violin, piano]      |
+----+---------+

How may I go about this?

changing the vec_of_strings to vec_of_vecs fails. I mean this

    let vec_of_vecs: Vec<Vec<String>> = vec![
        vec!["piano".to_string(), "guitar".to_string(), "drums".to_string()],
        vec!["violin".to_string(), "guitar".to_string()]
    ];

When used to create the batch like this

    let batch = RecordBatch::try_new(
        schema,
        vec![
            Arc::new(Int32Array::from_slice([1, 2])),
            Arc::new(StringArray::from(vec_of_vecs)),
        ],
    )?;

Fails to compile with the error

   |
80 |             Arc::new(StringArray::from(vec_of_vecs)),
  |                      ----------------- ^^^^^^^^^^^ the trait `From<Vec<Vec<std::string::String>>>` is not implemented for `GenericByteArray<GenericStringType<i32>>`
  |                      |
  |                      required by a bound introduced by this call
  |
  = help: the following other types implement trait `From<T>`:
            <GenericByteArray<GenericBinaryType<OffsetSize>> as From<GenericByteArray<GenericStringType<OffsetSize>>>>
            <GenericByteArray<GenericBinaryType<OffsetSize>> as From<Vec<&[u8]>>>
            <GenericByteArray<GenericBinaryType<OffsetSize>> as From<Vec<Option<&[u8]>>>>
            <GenericByteArray<GenericBinaryType<T>> as From<GenericListArray<T>>>
            <GenericByteArray<GenericStringType<OffsetSize>> as From<GenericByteArray<GenericBinaryType<OffsetSize>>>>
            <GenericByteArray<GenericStringType<OffsetSize>> as From<GenericListArray<OffsetSize>>>
            <GenericByteArray<GenericStringType<OffsetSize>> as From<Vec<&str>>>
            <GenericByteArray<GenericStringType<OffsetSize>> as From<Vec<Option<&str>>>>
          and 3 others

Any idea on how I may achieve the above?

[alamb]

How may I go about this?

I think you need to use a ListArray for this usecase

https://docs.rs/arrow/latest/arrow/array/type.ListArray.html

I recommend using a LIstBuilder like this:
https://docs.rs/arrow/latest/arrow/array/type.ListBuilder.html

Here is an example of use:

https://github.com/influxdata/influxdb_iox/blob/43078576b8b2896b3c148b21e6cc5c208a4bf3ea/flightsql/src/xdbc_type_info/mod.rs#L134C1-L135

(aka need to pass a StringBuilder to the ListBuilder

[finlaydotb]

Thanks for the pointer! I looked at the example, and the builder feels a bit intuitive :( I like to build a ListArray but I am using a builder that has this type signature ListBuilder<StringBuilder> - not sure I would have quickly gotten to this realisation by myself 😅

Is this the case because the example is building a ListArray of Strings?

[alamb]

Is this the case because the example is building a ListArray of Strings?

Yes

Thanks for the pointer! I looked at the example, and the builder feels a bit intuitive :( I like to build a ListArray but I am using a builder that has this type signature ListBuilder - not sure I would have quickly gotten to this realisation by myself 😅

I agree it could be improved -- if you have any specific suggestions I think they would be appreciated if you could file them in the upstream library https://github.com/apache/arrow-rs

[finlaydotb]

@alamb Thanks for the answer and pointing me to the example. It really helped.

By the way, where will you recommend to get more familiar with arrow/datafusion? Something more easier for a beginner to ease into?

[alamb]

Hi @finlaydotb -- I think https://arrow.apache.org/datafusion/user-guide/example-usage.html could help get you started and https://github.com/apache/arrow-datafusion/tree/main/datafusion-examples has a bunch of examples of doing various things with DataFusion

As for arrow, I think reading https://docs.rs/arrow/latest/arrow/index.html, starting at https://docs.rs/arrow/latest/arrow/index.html#columnar-format is the best introductory material we have at the moment for arrow-rs.

Given I no longer have the "new user perspective" I am not sure I understand what is obvious or not these days. Given you may have more of this perspective, perhaps you can help us identify what type of material you would find most helpful.