scala - Can only zip RDDs with same number of elements in each partition despite repartition -

-

i load dataset

val data = sc.textfile("/home/kybe/documents/datasets/img.csv",defp)

i want put index on data thus

val nb = data.count.toint val tozip = sc.parallelize(1 nb).repartition(data.getnumpartitions)  val res = tozip.zip(data)

unfortunately have following error

can zip rdds same number of elements in each partition

how can modify number of element partition if possible ?

why doesn't work?

the documentation zip() states:

zips rdd one, returning key-value pairs first element in each rdd, second element in each rdd, etc. assumes 2 rdds have same number of partitions , same number of elements in each partition (e.g. 1 made through map on other).

so need make sure meet 2 conditions:

  • both rdds have same number of partitions
  • respective partitions in rdds have same size

you making sure have same number of partitions repartition() spark doesn't guarantee have same distribution in each partition each rdd.

why that?

because there different types of rdds , of them have different partitioning strategies! example:

  • parallelcollectionrdd created when parallelise collection sc.parallelize(collection) see how many partitions there should be, check size of collection , calculate step size. i.e. have 15 elements in list , want 4 partitions, first 3 have 4 consecutive elements last 1 have remaining 3.
  • hadooprdd if remember correctly, 1 partition per file block. though using local file internally spark first creates kind of rdd when read local file , maps rdd since rdd pair rdd of <long, text> , want string :-)
  • etc.etc.

in example spark internally create different types of rdds (coalescedrdd , shuffledrdd) while doing repartitioning think got global idea different rdds have different partitioning strategies :-)

notice last part of zip() doc mentions map() operation. operation not repartition it's narrow transformation data it would guarantee both conditions.

solution

in simple example mentioned can data.zipwithindex. if need more complicated creating new rdd zip() should created map() mentioned above.


Comments

Post a Comment

Popular posts from this blog

java - Run spring boot application error: Cannot instantiate interface org.springframework.context.ApplicationListener -

-
i have spring project , try make use spring boot , run @ embadded tomcat following : https://spring.io/guides/gs/rest-service/ this application //@configuration //@enableaspectjautoproxy @springbootapplication @componentscan(basepackages = "gux.prome") public class application { public static void main(string[] args) { springapplication.run(application.class, args); } } if use maven command: mvn spring-boot:run , project starts fine, need debug , run main method @ intelij, exception occurs: exception in thread "main" java.lang.illegalargumentexception: cannot instantiate interface org.springframework.context.applicationlistener : org.springframework.boot.logging.classpathloggingapplicationlistener @ org.springframework.boot.springapplication.createspringfactoriesinstances(springapplication.java:414) @ org.springframework.boot.springapplication.getspringfactoriesinstances(springapplication.java:394) @ org.springframework.bo...
Read more

python - pip wont install .WHL files -

-
when try install .whl gohike typing cmd c:\users\owner\downloads>pip install ta_lib-0.4.9-cp27-none-win_amd64.whl i print out: `processing c:\users\owner\downloads\ta_lib-0.4.9-cp27-none-win_amd64.whl complete output command python setup.py egg_info: traceback <most recent call last>: file "<string>", line 1, in <module> ioerror: [errno 2] no such file or directory:'c:\\users\\owner\\appdata\\lo cal\\temp\\pip-9gwk2c-build\\setup.py' ----------------------------------------- command "python setup.py egg_info" failed error code 1 in c:\users\owner\ap pdata\local\temp\pip-9gwk2c-build\` the part between pip- , -build in pip-9gwk2c-build different on same file. i asked python 2.7.11 pip 8.0.3 can't install gohike whl of ta-lib 0.4.9 few days ago , decided ask differently. can me! you have outdated pip not understand wheels. wheels not supposed have setup.py inside. try upgrading pip pip in...
Read more

Excel VBA "Microsoft Windows Common Controls 6.0 (SP6)" Location Changes -

-
i have excel workbook used generate quotations customers. there main page user inputs data, , there buttons generate various pdfs , send them customer. however, 1 user consistently getting error preventing generating , sending pdfs. i beleive "microsoft windows common controls 6.0 (sp6)". it's installed on users' computers. on mine, location "c:\windows\syswow64\mscomctl.ocx" on others "c:\windows\system32\mscomctl.ocx" the user experiencing issues on 32-bit machine, "mscomctl.ocx" file should in system32. have manually put in there, , pointed common controls reference in vba it. issue seems keep reverting looking in syswow64. since folder doesn't exist, can't find it, , can't produce pdfs. i thought perhaps issue user running older version of office (2007) there users on 2013 , 2016. thought perhaps mixture of 64-bit , 32-bit users. user gets sent workbook other users (as quotation process goes further, gets passed ...
Read more