Newest 'scala' Questions - Stack Overflow

Questions tagged [scala]

Scala is a general-purpose programming language principally targeting the Java Virtual Machine. Designed to express common programming patterns in a concise, elegant, and type-safe way, it fuses both imperative and functional programming styles. Its key features are: an advanced static type system with type inference; function types; pattern-matching; implicit parameters and conversions; operator overloading; full interoperability with Java; concurrency

0
votes
1answer
19 views

Conditional termination from foldLeft iteration

Limit number of iteration in foldLeft. I would like to terminate iteration of foldLeft based on certain condition. Condition is evaluated from different variable not from list element. I tried using ...
1
vote
1answer
12 views

Does use of Future.successful and Future.failed request a thread from the thread pool managed by the execution context

As we all know, use of Future in Scala requires declaration of an execution context, which creates a thread pool that is used to run the code that appears in Future. My question is if use of Future....
-1
votes
0answers
9 views

Task not serializable when iterating through dataframe, scala

Below is my code and when I try to iterate through each row: val df: DataFrame = sqlContext.read .format("com.databricks.spark.csv") .option("header", true) // Use first line of all files as ...
0
votes
0answers
12 views

Slick: Composed, optional columns to composed optional type

This is given: case class Money(value: BigDecimal, currency: Currency) trait Currency case class EUR ( ... ) extends Currency ... and so on I want to model a optional Money type which contains a ...
0
votes
0answers
9 views

Reading multiple files with akka streams in scala

I'am trying to read multiple files with akka streams and put result in a list. I can read one file with no problem. the return type is Future[Seq[String]]. problem is processing the sequence inside ...
3
votes
1answer
43 views

Is it possible to simplify this scala match statement using @ syntax?

Is it possible to simplify the following match statement using @ syntax? foo match { case f: Foo => y(f) case f if forceY => y(f) case _ => x } where forceY is a boolean. I've tried ...
0
votes
0answers
26 views

How to remove all rows that contains a NULL?

I have a small problem. I would like to delete any row that contains 'NULL'. This is my input file: matricule,dateins,cycle,specialite,bourse,sport 0000000001,1999-11-22,Master,IC,Non,Non 0000000002,...
0
votes
3answers
19 views

Create custom json definition in play scala

I'm new to Scala and I'm learning Scala and Play Framework: I'm trying to dynamically create a Json with play/scala starting from a sequence of data named "tables" by using Map(...), List(...) and ...
0
votes
1answer
34 views

Does flatmap give better performance than filter+map?

I have a quite a big dataset (100 million+ records with 100's of columns) that I am processing with spark. I am reading the data into a spark dataset and I want to filter this dataset and map a subset ...
1
vote
1answer
43 views

Why does it not recognize asScala method?

I have the following code, that does not compile: import akka.NotUsed import akka.actor.typed.Behavior import akka.actor.typed.scaladsl.Behaviors import akka.pattern.FutureRef import akka.stream....
0
votes
2answers
24 views

Converting dataframe to xml in spark throws NPE in StaxXML

I am reading an xml file using sparkSession based on a given rowTag. The resultant dataFrame obtained needs to be converted to xml file. Below is the code I am trying out: val sparkSession = ...
0
votes
1answer
11 views

Spring MVC Scala App just returns index.html page rest of the routes doesn't work

I am trying to create a simple spring MVC app in Scala I did define my methods in the controller to bring back html pages based on name from resources folder but it just always brings back just index ...
0
votes
1answer
13 views

using external ivy for sbt build

At my work place we are starting to use Scala. For dependencies and building we use an ivy.xml file. I am struggling to get the build.sbt to work with the ivy.xml. https://www.scala-sbt.org/0.13/docs/...
0
votes
0answers
10 views

Jackson doesn't serialize field defined by `def`

Following the approach in Easy JSON (un)marshalling in Scala with Jackson, I have defined a JsonUtil class: import com.fasterxml.jackson.databind.{DeserializationFeature, JsonNode, ObjectMapper} ...
-2
votes
0answers
14 views

I want to read multiple text files and save each of the files into multiple dataframe using spark scala

"Reading multiple text files with dynamic schema and saving each of the files as dataframes individually using spark scala.The schema can be different and it should be automatically inferred " I am ...
0
votes
1answer
36 views

MapReduce functions

I am working on a spark project on eclipse IDE using scala i would like some help with this MapReduce problem Map function: remove column 'sport' and 'bourse' delete any row that has 'NULL' Add a ...
0
votes
1answer
13 views

How to query flink's queryable state

I am using flink 1.8.0 and I am trying to query my job state. val descriptor = new ValueStateDescriptor("myState", Types.CASE_CLASS[Foo]) descriptor.setQueryable("my-queryable-State") I used ...
1
vote
0answers
26 views

Scala/ Java, Summing up two arrays columns in a Dataset

I have the following Dataset, currently using the UDF to sum up the arrays. The performance is slow since I'm using an UDF. Instead I'm looking to use High order functions or spark.sql.functions or ...
-1
votes
3answers
33 views

String interpolation for array using map in scala

I have an array = ("1","2","3") and I also have a string that I want to format: "item = {} found" I want to get an array: ("item = 1 found", "item = 2 found", "item = 3 found")
0
votes
0answers
28 views

How can I add an Index grouped by ID in Dataframe Spark

I have this Dataframe +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-...
1
vote
0answers
10 views

scala elasticsearch using high level rest java client: regexp and scroll

I am using scala (2.12) to query the Elasticsearch (6.5) using the High level java rest client(6.5). libraryDependencies += "org.elasticsearch" % "elasticsearch" % "6.5.4" libraryDependencies += "...
3
votes
1answer
45 views

overridable default type parameter in Scala

I am trying to expose a class parametrized with a type alias: trait Tr[T] { var x: T = _ } abstract class Foo { type MyParameter class SomeClass extends SomeOtherClass with Tr[MyParameter] } ...
0
votes
0answers
17 views

How to fix compilation errors when bumping from sbt 1.2.4 to 1.2.5

When bumping sbt from 1.2.4 to 1.2.5, some compilation errors appear Any idea ? [error] This symbol is required by 'class xxx.RedisClients'. [error] Make sure that type LazyLogging is in your ...
0
votes
0answers
24 views

How to make more partitions than the threads that I have?

I have a cluster with the following characteristics: 4 nodes 8 cores per node 8GB per node When I run my program I do it with the following characteristics: --deploy-mode cluster --executor-...
0
votes
0answers
11 views

Scala generic query predicate generation DSL (possibly similar to AlaSQL in javascript)?

I'm building a generic API that provides access to a ton of different data sets, and I would like to avoid creating specific query API's for each data type. I.e. for the Users endpoint needing to ...
1
vote
0answers
24 views

How to collect a single row dataframe and use fields as constants

I'm trying to read a single row from a Hive table and use its fields as constants in the rest of my Spark application. object IniConstHive extends Serializable { val techTbl: DataFrame = spark....
0
votes
1answer
13 views

No Instance of play.api.libs.json.Format is availablefor scala.collection.immutable.List[Item]

Note: I'm very new to PlayFramework. I'm trying to serialise an item object into a JSON String. I receive the following error: No instance of play.api.libs.json.Format is available for scala....
2
votes
1answer
30 views

HTTP URL Address Encoding in Scala/Java

I want a function that loads the contents of a url address, but I do not know in advance if the url is encoded or not. Is there a library function that solves that? One possible soluction would be ...
1
vote
2answers
16 views

Spark does not use correct configuration from core-site.xml

When I try to read a parquet file from a specified location like /test with spark.read.parquet() i get an error saying file://test does not exist. When I add the core-site.xml as a resource in code ...
1
vote
1answer
32 views

Calling main methods in other objects in Scala

Is it possible to call a main method in one object from a main method in another? I have the following classes and was wondering how to call two separate main methods within one program run: object ...
1
vote
1answer
31 views

how can an example have two packages with same name

Referring to the example at https://www.scala-lang.org/files/archive/spec/2.11/02-identifiers-names-and-scopes.html, I notice that the package P is defined twice in the example. Isn't the example ...
1
vote
0answers
33 views

Online IDE to run the following scala code

I want to test the following scala code. I want to use an online editor. I tried scalafiddle and jdoodle but both can't identify the work package. Is there an online IDE where I can run the following ...
0
votes
1answer
24 views

Subscribing to an Akka stream, reading files

I'm trying to read a file in Scala with Akka Streams, I want to put the result in a list. I tried the following code, and the list gets incremented with new values inside the sink, but outside I get ...
2
votes
1answer
34 views

Retry with resilience4j doesn't work with some exceptions

I am using the resilience4j library to retry some code, I have the following code below, I expect it to run 4 times. If I throw IllegalArgumentException it works but if I throw ConnectException it ...
1
vote
3answers
70 views

Converting object of type Any into myClass, by passing the myClass as parameter

So I have a class: case class Document (idx: String, name: String, code: String) Due to some transformations, an object which was initially created as Document, now becomes of type Any. val doc = ...
2
votes
0answers
25 views

Spark 2.4 CSV Load Issue with option “nullvalue”

We were using Spark 2.3 before, now we're on 2.4: Spark version 2.4.0 Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212) We had a piece of code running in production that ...
-1
votes
1answer
29 views

rdd.map invoking a function twice rather than once

I am a newbie to scala programming and I am currently working with RDD. I am trying to pass an RDD to a function and would like the function to return so that I can store it to a new RDD. For the ...
1
vote
2answers
43 views

Why do I have to import Ordering.Implicits?

In he code below I get a compilation error unless I import Ordering.Implicits. Why is that import necessary? Is there a way around that? Am I doing something completely off-base? Error:(15, 38) value ...
1
vote
3answers
60 views

How to reference a variable when multiple cases are combined?

Is there a way to give a reference name to a variable when multiple cases are combined using match in Scala? Code: Assume Gender enum has three possible values like male, female and other. (...
0
votes
1answer
24 views

pros and cons of Avro - JSON / binary serialization

I've recently started working with Apache-Avro and would like to serialize my Kafka Topics. Now I've read that Avro offers both Json and binary serialization. Are there any pros and cons? Is not the ...
0
votes
1answer
45 views

How do I split contents of a list that is part of a dataframe?

I've got a dataframe where each row is a list of pairs. For instance, I've got a dataframe df that looks something like this: id | pairs 1 | (a, b), (b, c), (a, c) 2 | (a, b), (b, d) 3 | (b, c) ...
-1
votes
0answers
28 views

How to connect Html and Scala using Playframework or Spray in intellij?

I want to upload images through the html page and load into the server using scala code(i am doing a project using Apache Spark).I am using Intellij as IDE.I'm trying playframework and spray but it ...
1
vote
1answer
42 views

How to convert a space delimited file to a CSV file in Scalar spark?

I has a CSV file. This is my Input: a _ \_ \ b_c b\_c " Now, I want to convert a space delimited file to a CSV file. What should I do? Fields not specified are considered "String 0" and are not ...
1
vote
1answer
23 views

Scala with Spring Boot strange behaviour: hangs and no exception on input failure

I am using the Scala. I am not so familiar with Scala, but at least it worked to use the jackson JSON parser. Now, I have a very strange problem. If the input JSON String has a syntax error, the ...
1
vote
2answers
48 views

Spark creating a new column based on a mapped value of an existing column

I am trying to map the values of one column in my dataframe to a new value and put it into a new column using a UDF, but I am unable to get the UDF to accept a parameter that isn't also a column. For ...
0
votes
2answers
61 views

How do I access a nested function in a scala object

I'm trying to access a nested function. I am able to access func1 but I'm running into issues when trying to access func2. //attempting to call methods in different class val t1 = test t1.func1() t1....
1
vote
2answers
30 views

Execute operation on Future's value for its side effects, discarding result and retaining original value, but retaining order of operation

Say I have the following operations that must proceed in order: Get blog post Post analytics Forward blog post In code it may look like this: val blogPostFut: Future[BlogPost] = blogService.getPost(...
0
votes
0answers
8 views

I'm having a error while building a Cube in Apache Kylin

See the Logs in the Step one: java.io.IOException: OS command error exit with return code: 127, error message: Logging initialized using configuration in jar:file:/home/jefferson/Developer/apache-...
0
votes
1answer
34 views

How to concatenate list of strings with map reduce in scala

I have a list of list of strings and I want to concatenate all the unique strings into a single (delimited space) string, something that flatMap allows one to do. However I am confused about the ...
0
votes
1answer
15 views

SBT: “value map is not a member of (sbt.TaskKey[sbt.Tests.Output], sbt.TaskKey[sbt.Tests.Output]) ” when updating from 0.13.8 to 1.x.x

I've below sbt-mult-jvm plugin which are working fine on sbt version 0.13.18. import com.typesafe.sbt.SbtMultiJvm import com.typesafe.sbt.SbtMultiJvm.MultiJvmKeys.MultiJvm lazy val ...