在Java（不是Scala！）Spark 3.0.1有一个JavaRDD实例对象邻居IdsRDD，它的类型是JavaRDD
我与JavaRDD生成相关的部分代码如下：
GraphOps<String, String> graphOps = new GraphOps<>(graph, stringTag, stringTag);
JavaRDD<Tuple2<Object, long[]>> neighborIdsRDD = graphOps.collectNeighborIds(EdgeDirection.Either()).toJavaRDD();
我不得不使用toJavaRDD（）获取JavaRDD，因为Collection ectNighborIds返回一个org. apache.park.graph x.VertexRDD
但是，在我的应用程序的其余部分中，我需要一个Spark数据集
获得JavaRDD的可能性和最佳方法是什么
我根据注释调整了代码：
GraphOps<String, String> graphOps = new GraphOps<>(graph, stringTag, stringTag);
JavaRDD<Tuple2<Object, long[]>> neighborIdsRDD = graphOps.collectNeighborIds(EdgeDirection.Either()).toJavaRDD();
System.out.println("VertexRDD neighborIdsRDD is:");
for (int i = 0; i < neighborIdsRDD.collect().size(); i++) {
System.out.println(
((Tuple2<Object, long[]>) neighborIdsRDD.collect().get(i))._1() + " -- " +
Arrays.toString(((Tuple2<Object, long[]>) neighborIdsRDD.collect().get(i))._2())
);
}
Dataset<Row> dr = spark_session.createDataFrame(neighborIdsRDD.rdd(), Tuple2.class);
System.out.println("converted Dataset<Row> is:");
dr.show();
但我得到一个空的数据集，如下所示：
VertexRDD neighborIdsRDD is:
4 -- [3]
1 -- [2, 3]
5 -- [3, 2]
2 -- [1, 3, 5]
3 -- [1, 2, 5, 4]
converted Dataset<Row> is:
++
||
++
||
||
||
||
||
++
我和你的情况一样，但幸运的是，我找到了一个取回数据帧的解决方案。
解决方案代码在步骤[1]、[2]和[3]中注释。
GraphOps<String, String> graphOps = new GraphOps<>(graph, stringTag, stringTag);
System.out.println("VertexRDD neighborIdsRDD is:");
JavaRDD<Tuple2<Object, long[]>> neighborIdsRDD = graphOps.collectNeighborIds(EdgeDirection.Either()).toJavaRDD();
for (int i = 0; i < neighborIdsRDD.collect().size(); i++) {
System.out.println(
((Tuple2<Object, long[]>) neighborIdsRDD.collect().get(i))._1() + " -- " +
Arrays.toString(((Tuple2<Object, long[]>) neighborIdsRDD.collect().get(i))._2())
);
}
// [1] Define encoding schema
StructType graphStruct = new StructType(new StructField[]{
new StructField("father", DataTypes.LongType, false, Metadata.empty()),
new StructField("children", DataTypes.createArrayType(DataTypes.LongType), false, Metadata.empty()),
});
// [2] Build a JavaRDD<Row> from a JavaRDD<Tuple2<Object,long[]>>
JavaRDD<Row> dr = neighborIdsRDD.map(tupla -> RowFactory.create(tupla._1(), tupla._2()));
// [3] Finally build the reqired Dataframe<Row>
Dataset<Row> dsr = spark_session.createDataFrame(dr.rdd(), graphStruct);
System.out.println("DATASET IS:");
dsr.show();
打印输出：
VertexRDD neighborIdsRDD is:
4 -- [3]
1 -- [2, 3]
5 -- [3, 2]
2 -- [1, 3, 5]
3 -- [1, 2, 5, 4]
DATASET IS:
+------+------------+
|father| children|
+------+------------+
| 4| [3]|
| 1| [2, 3]|
| 5| [3, 2]|
| 2| [1, 3, 5]|
| 3|[1, 2, 5, 4]|
+------+------------+
